Leveraging a rolling cross-section design in panel studies to investigate police legitimacy

Monica Gerber and Cristóbal Moya



London School of Economics and Political Science

May 22, 2023

Today

  1. Motivation
  2. ‘Unexpected event during survey’ design
  3. The rolling cross-section design
  4. This Study: a rolling cross-section panel design
  5. Implementation challenges

Motivation

Confirmatory research

  • Inferences about changes in police legitimacy
  • Panel study
    • Interview individuals at two or more points in time

Exploratory research

  • Unexpected events relevant to police legitimacy
  • Rolling cross-sectional design (RCS)
    • Randomized controlled interviewing over time

‘Unexpected event during survey’ design

“research design that exploits the occurrence of an unexpected event during the fieldwork of a public opinion survey to estimate its causal effect on a relevant outcome by comparing responses of the individuals interviewed before the event \(t_i<t_e\) (control group) to those of respondents interviewed after the event \(t_i>t_e\) (treatment group).” (Muñoz et al., 2020, p. 187)

  • Quasi-experiment: Effects of naturally occurring events that cannot be manipulated through controlled experiments
  • Assumption that interview time is independent from the time when the event occurs

The RCS design

An ‘expected-unexpected’ event during survey design

“the design is just a standard cross-section, but the day on which a respondent is interviewed is chosen randomly” (Johnston & Brady, 2022, p. 283)

“The data collected in an RCS allow a fine-grained analysis of dynamic phenomena of public opinion” (Lutz et al., 2013, p. 3)

  • Temporal heterogeneity as an opportunity to move closer to causal inference (Johnston & Brady, 2002)

The RCS design

The ‘expected-unexpected’ event during survey design


Steps (Kenski et al., 2010):

  1. Total number of interviews is established
  2. Full sample is randomly divided into multiple replicates
  3. Replicates assigned to be interviewed at different intervals of times
  4. The same protocol is followed for all replicates

The RCS design

Example

The RCS design

Assumptions


1. Temporal ignorability: Moment of the interview should be as good as random

  1. Excludability: Timing of the interview \(t\) affects \(Y\) only through event \(T\)

  2. Compliance: assignment to the treatment (\(t_i>t_e\)) corresponds to actually receiving treatment (\(T=1\))

This Study

Police legitimacy


  • Police work requires public support to be able to function in a democratic context

  • Beliefs that the police act in procedurally just ways and are legitimate facilitate compliance and cooperation with authorities (Jackson et al., 2012; Tyler, 2006, 2011)

  • A socially legitimate police force should require little violence to confront opposition

  • However, the world has seen many cases of violent demonstrations, police violence and human rights violations

This Study

The Chilean case


  • Chile’s main police force, the Carabineros, was among the most trusted institutions (Dammert, 2019)

  • Yet, recent events of corruption and excessive use of force during the social uprising in October 2019 have produced a serious loss of trust (Gerber et al., 2023)

  • Crisis of legitimacy: proposals of “deep reform” and even the dissolution of the institution

  • Need to understand relationship between police and citizens in Chile during the next crucial years

This Study

Aims


  • Confirmatory:
    • Track over time people’s perceived police legitimacy and motivational postures towards the police, and examine their relationships with perceived procedural justice and fear of the police in Chile
  • Exploratory:
    • Determine the effects of potential unexpected events (such as police violence, corruption scandals or reforms) on perceptions of legitimacy, procedural justice and fear of the police in Chile.

This Study

A RCS panel design


  • Probability-based online panel of the population in Chile (18-65 years old)

  • Panel study with 10 waves starting in October 2023

  • Face-to-face recruitment with probability-based sampling design

  • Sample size

    • Expected sample in contact survey: n = 5,300
    • Expected sample in Wave 1: n = 1,500
  • Possibility to combine with non-probabilistic sample to increase sample size

This Study

A RCS panel design


This Study

A RCS panel design


This Study

A RCS panel design


This Study

Identify unexpected event with RCS design

This Study

Identify unexpected event with panel design

This Study

Identify unexpected events with RCS panel design

Contribution of the RCS panel design

  • Facilitates the causal identification of unexpected events
    • During waves
    • Between waves
  • Potential to identify interactions between
    • Unexpected events within and between waves
    • Unexpected events with main study hypotheses
  • But at what cost?

Implementation challenges


Event happens on... Event Power
Not treated Treated Small effect Medium effect Large effect
Day 4 129 1329 0.58 1.00 1.00
Day 7 257 1200 0.83 1.00 1.00
Day 10 386 1071 0.92 1.00 1.00
Day 13 514 943 0.95 1.00 1.00
Day 17 686 771 0.97 1.00 1.00

Implementation challenges

Thank you!

Contact: